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Background / Context: 

Description of prior research and its intellectual context. 



A common setting in educational research consists of a randomized intervention at the sehool 
level, a mediator of interest at the classroom or teacher level and an outcome of interest at the 
child level. A eommon approach to addressing mediation in sueh settings consists of regressing 
the outcome on the treatment with and without the mediator variable. This approach to mediation 
analysis is subject to several limitations. First, the approaeh ignores seleetion into mediator 
levels; although the treatment is randomized, the mediator is not and thus analyses ignoring this 
selection issue are subject to potentially severe biases due to confounding (Judd and Kenny, 

1981; Robins and Greenland, 1992; Pearl, 2001). The seeond issue with the standard regression 
approaeh is that potential interaetion between the effects of treatment and the mediator on the 
outcome are typically ignored. Recent literature on causal inference has made elear that 
mediation analysis becomes considerably more complex when sueh interactions are present 
(Pearl, 2001). A third issue with the standard regression approach is that it ignores issues of 
interference and spill-over effects. Child level outeomes may depend not only on the 
characteristies of the child’s own classroom but also on the characteristics of other classrooms 
because of soeial interaetions among children from different classrooms. The issue is referred to 
as one of interferenee between units in the statisties literature (Cox, 1958). No interference 
between units is a component of Rubin’s Stable Unit Treatment Value Assumption or SUTVA 
(Rubin, 1980, 1986). The assumption will be violated in settings in which social interactions 
allow one individual’s exposure to affect the outeomes of other individuals. Such interference is 
part of the theoretical rationale of the 4Rs program whieh foeuses on bringing together 
educators’ collective efforts within a school. Analyses of causal effects are considerably more 
complex in the face of such interference. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 



In this paper we extend reeent work on mediation in a multilevel setting (VanderWeele, 2010) 
and on eausal inferenee under interferenee among units (Hong and Raudenbush, 2006; Hudgens 
and Halloran, 2008; Rosenbaum, 2007; Sobel, 2006) to develop a template for the mediation 
analysis of group randomized edueational interventions. The present work will contribute to the 
literature on interference, in particular on interference in the context of mediation analysis. We 
will show that not only does the total effect of the intervention deeompose into a direct effect and 
an indirect effect mediated through classroom quality but also that the indireet effeet itself 
deeomposes into an effect mediated through the quality of a child’s own classroom and a 
spillover effeet from the quality of the other elassrooms at a school. We will give some results 
for the identifieation of these direct, indirect and spillover effects and consider the consequences 
of ignoring interferenee when it is in fact present. We will then analyze the effects of the 
Reading, Writing, Respect and Resolution (4Rs) intervention in a group randomized trial. 



Significance / Novelty of study: 

Description of what is missing in previous work and the contribution the study makes. 
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Peer influenee and soeial interaetions ean give rise to spillover effeets in whieh eharaeteristies of 
one individual unit may affect outcomes of other individual units. Evaluators who choose groups 
rather than individuals as experimental units in group randomized trials often anticipate that the 
desirable changes in targeted social behaviors will be reinforced through interference among 
individuals in a group exposed to the same treatment. Failure to account for such spillover effects 
can result in bias and problems with interpretation. Using a counterfactual conceptualization of 
direct, indirect and spillover effects, we provide a framework that can accommodate issues of 
mediation and spillover effects in group randomized trials. 

Statistical, Measurement, or Econometric Model: 

Description of the proposed new methods or novel applications of existing methods. 



In this paper, we are interested in the extent to which the effect of the 4Rs intervention on child 
outcomes is mediated by classroom quality. Let denote the school-wide randomized treatment 
for school k{\ for the 4Rs intervention; 0 for control). Let Mjk denote the classroom level 
mediator for classroom j in school k. In the 4Rs intervention study the mediator of interest is a 
continuous measure of classroom quality. Let Jk denote the number of classrooms in school k. 

Let Yijk denote the child-level outcome for child i in classroom j and school k. In the 4Rs study 
this outcome is a continuous score measuring depressive symptoms. Let Yijk{tk) denote the 
potential or counterfactual outcome that child i in classroom j and school k would have obtained 
if the school-level treatment, Tk, were set to tu. Similarly, let Mjk{tk) denote the potential or 
counterfactual mediator that classroom j in school k would have obtained if the school-level 
treatment, Tk, were set to tk. We assume that children do not change schools as a result of the 
treatment to which a particular school is assigned. Hong and Raudenbush (2006) referred to this 
assumption as that of "intact clusters." We also assume that there is no interference between 
schools (i.e. that the treatment received at one school does not affect the outcomes of the children 
at any other schools). To incorporate within-school interference into our potential outcomes 
notation, we let rtijk, m.^^:) denote the counterfactual outcome that child i in classroom j and 
school k would have obtained if the school-level treatment in school k were set to tk, if the quality 
in classroom j of school k were set to nijk and if the quality of all other classrooms in school k 
were set to the vector m.jk = {m\k, ..., nij.ik, mj+ik, ..., mjtk). Following Hong and Raudenbush 
(2006) and Hudgens and Halloran (2008), we assume that the potential outcome Ty^(h, nijk, m.jk) 
depends on m.^^: through some scalar function G{m.jk) of m.jk so that we may express the potential 
outcome as Tyi(h, mjk, G{m.jk)). For example, Gfm.jk) may denote the average quality for all 
classrooms in school k other than classroom j. Here we let Yijk{t, m, g) denote the outcome for 
child i in classroom j and school k if the school received treatment t, the child’s classroom had 
quality m, and the scalar function of the quality of other classrooms, Gfm.jk), took the value g. 

The causal contrast E[Yijk{\, m, g) - Tyi(0, m, g)] captures the direct effect of the 4Rs program 
but also intervening to fix the quality of the child’s own classroom to level m and intervening to 
fix the average quality of other classrooms to g. This quantity is referred to as a controlled direct 
effect of treatment. Likewise the contrast E{Yijk{t, m, g) - Yijk{t, m*, g)] could be used to assess 
the effect of a child’s own classroom quality (comparing levels m and m*) on a child’s outcome 
and to examine whether the contrast varies with t or g. Similarly, the contrast E\Yijk{t, m, g) - 
Yijk{t, m, g*)] could be used to assess the spillover effect of the quality of classrooms other than 
the child’s own and whether the contrast varies with t or m. 
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When the classroom quality mediators are set to the levels they would have been at under the 
control condition, the natural direct effect is defined as E{Yijk (1, Mjk (0),G(M.jjt(0))) - Yijk (0, Mjk 

(0) , G(M.ji:(0)))]. We can also define a natural indirect effect as E[Yijk (1, Mjk (l),G(M.j^:(l))) - 
Yijk (1, Mjk (0),G(M.ji(0)))]. As in the case of non-clustered treatments without interference 
(Pearl, 2001), the total effect of the intervention on the outcome E[Yijk (1) - Yijk (0)] decomposes 
into natural direct and indirect effects. The decomposition will hold even if there are interactions 
between the effects of the treatment and the mediator on the outcome. The natural indirect effect 
further decomposes into a within-classroom mediated effect E[Yijk (1, Mjk (l),G(M.jA:(0))) - Yijk 
(1, Mjk (0),G(M.y/t(0)))] and a spillover mediated effect £[% (1, Mjk (l),G(M.y^(l))) - Yijk (1, Mjk 

(1) ,G(M,,(0)))]. 

We will let X,y^:, W/yi, and denote child-level, class-level, and school-level baseline covariates, 
respectively. We will use to denote the vector of child-level baseline covariates for children 
in school k other than child i in classroom j, W.jk to denote classroom-level baseline covariates 
for classrooms in school k other than classroom j. We will consider certain functions of the 
baseline covariates of other children in the classroom (or even at the school), hi(X.,y^:), and of 
baseline covariates of classrooms other than a child’s own h 2 (W.y^:). To simplify notation we let 

Lijk = (Xijk, Wjk, \k, hi(X.ijk), h2(W.jk)). 



Usefulness / Applicability of Method: 

Demonstration of the usefulness of the proposed methods using hypothetical or real data. 



We present four identification results, one for controlled direct effects, one for natural direct and 
indirect effects, one for the spillover and within-classroom mediated effects, and one for the 
consequences of ignoring interference when it is in fact present. For sets of random variables A, 
B, and C, we will use A B | C to represent that A is independent of B conditional on C. 

Theorem 1 . If for all t, m, g we have that Yijkft, m, g)-^Tk\ and that Yijk{t, m, g) {Mjk, G(M. 
jk)} 1 Yk, Lijk, then we can identify the controlled direct effect of the treatment and that of each 
mediator. 

Theorem 2. If in addition to the assumptions stated in Theorem 1, we also have that [Mjkff), 
G(M.jk(t))} Tk I Lijk and that for all t, t*, m, g, Yijk(t, m, g) -*L {Mjk(t*), G(M.y^(t*))} | Lijk, then 
we can identify the natural direct effect and the natural indirect effect. 

Theorem 3. If in addition to the assumptions stated in Theorems 1 and 2, we also have that for t' 
t* , Mjkit ) G(M.j^(t*)) I Lijk, then we can identify the within-class mediated effect and the 

spillover mediated effect. 

Theorem 4. Suppose that the assumptions stated in Theorems 1, 2, and 3 hold. And suppose we 
also require that for all t, Mjkit) G(M.jk{t)) \ Lyk, then we can ignore interference while still 
obtaining an estimate of the within-classroom mediated effect and obtaining the sum of a 
spillover mediated effect and the actual natural direct effect. However, even if all the above 
assumptions hold, if the substantive question of interest is whether classroom quality mediates 
the effect of treatment, ignoring interference would lead to an underestimate of the actual 
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importance of classroom quality since it will not include an assessment of the effect mediated 
through the quality of other classrooms. 

Research Design: 

Description of research design (e.g., qualitative case study, quasi-experimental design, secondary analysis, analytic 
essay, randomized field trial). 

(May not be applicable for Methods submissions) 

The 4Rs program is a school-based intervention in literacy development, conflict resolution, and 
intergroup understanding. The study (Jones, Brown, & Aber, in press) involved a 3-year, 6-wave 
longitudinal experimental design with measurements in the fall and spring semester of each year. 
The eighteen New York City elementary schools in the study were fairly representative of the 
demographic characteristics of New York City schools and included 923 students in 82 
classrooms. The schools were pair matched based on twenty school characteristics including 
size, reading achievement, race/ethnic composition, mobility/two-year stability, school lunch 
receipt, expenditures, attendance and organizational readiness. Within each pair, schools were 
randomly assigned to either the 4Rs treatment or the control group. The intervention was 
implemented school-wide from grades K-6 for 3 years. All 3rd grade children in each school 
were followed over three years through 5th grade. In the application here, we will consider the 
first year of the study for the children beginning in third grade. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

(May not be applicable for Methods submissions) 

Classroom quality was measured in the spring semester using the CLASS scoring system (Pianta, 
La Paro, and Hamre, 2005) which assesses instructional support, emotional support, and 
organizational climate with an overall score between 1 and 7. We dichotomized this measure 
using 4.4 as the cutoff The child-level outcome was depressive symptoms scored on a scale of 0 
to 1. Covariates in the model were chosen based on prior empirical work (Brown, Jones, 
LaRusso, & Aber, 2010). The covariates were at least marginally predictive of either the 
outcome or the mediator. The models also included pair fixed effects to control for school-level 
factors. We fitted a multilevel model for the effect of treatment on the depressive symptoms, a 
multilevel model for the effect of treatment on class quality, and finally a multilevel model for 
the effects of treatment, classroom quality, and quality of other classrooms on depressive 
symptoms with the interactions between these variables saturated. The parameter values and 
model-based standard errors were estimated via maximum likelihood in HLM 6.0. 

Findings / Results: 

Description of the main findings with specific details. 

(May not be applicable for Methods submissions) 

The estimated treatment effect of the 4Rs intervention on depressive symptoms is -0.052 {s.e. = 
0.023, t = -2.29, p = 0.05), suggesting a marginally significant effect of the treatment in reducing 
child depressive symptoms. The estimated treatment effect of the 4Rs intervention on classroom 
quality is 0.45 {s.e. = 0.20, t = 2.2%, p = 0.05). In the control schools it appears that depressive 
symptoms are highest for children in classrooms in which the quality of the child’s own 
classroom is low but the quality of other classrooms at the school is relatively high. Apparently it 
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is also only in this type of classrooms that the 4Rs intervention has a statistically significant 
direct effect on depressive symptoms (see Table 1). However, a % test failed to reject the null 
hypothesis of no interaction of treatment with either a child’s own classroom quality or the 
quality of other classrooms. We then obtained an estimate of the overall controlled direct effect 
of treatment of -0.058 ip = 0.025). If we take the x test result as an indication that the 
assumptions in Theorem 2 holds, then this estimate would also correspond to the natural direct 
effect. We could obtain a natural indirect effect by subtracting the natural direct effect from the 
total effect of treatment which would give -0.052 — (-0.058) = 0.006. This provides very little 
evidence that any of the effect of the 4Rs intervention on depressive symptoms is mediated 
through either the quality of a child’s own classroom or the quality of the classes other than the 
child’s own. Because this particular evaluation of the 4Rs intervention was powered to be able to 
assess only the total effect, not direct and indirect effects, our results here are at best suggestive. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 



In this paper we have made a number of contributions to allow researchers to address questions 
of mediation and spillover effects in group randomized trials. The approach we have described 
here constitutes an advance over the standard approach to estimating direct and indirect effects 
that is often used in group-randomized trials. Specifically, our approach (i) makes explicit the 
assumptions required for identification that will be important in study design and data analysis of 
group randomized trials, (ii) accommodates possible interactions that may be present, (iii) allows 
for interference between individuals in different clusters (e.g., classrooms in the 4Rs evaluation), 
and (4) allows for the definition, identification, and estimation of spillover effects. In particular, 
by relaxing the no-interference assumption, we have been able to investigate spillover effects 
that will often be of substantive and theoretical interests. Interference is not simply a problem 
that must be dealt with but in fact gives rise to research questions about spillover effects that are 
of interest in their own right. In addition, we have provided an analysis of the mediation and 
spillover effects in the 4Rs evaluation. The chief limitations of the analysis are: (i) a relatively 
large sample size may be required to draw reliable inferences about mediation and spillover 
effects; and (ii) relatively strong identification assumptions are required to empirically estimate 
these effects from data. 

The approach that we have presented here could be extended in a number of directions. First, 
future work could consider accommodating longitudinal settings as the mediator and outcome 
changes over time. Second, work has been done on using weighting techniques (van der Laan 
and Petersen, 2008; VanderWeele, 2009; Hong, 2010) rather than regression to address 
confounding control in questions of mediation analysis; future research could attempt to extend 
these weighting techniques to estimate and distinguish spillover mediated effects and within- 
classroom mediated effects. Third, further research could develop sensitivity analysis techniques 
to assess the extent to which an unobserved variable affecting both the mediator and the outcome 
(and thus giving rise to confounding of the effects of both the mediator in a child’s own 
classroom and that of the mediator in the other classrooms) might invalidate the inference about 
direct, indirect and spillover effects. 
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Table 1 

Controlled Direet Effeets of 4Rs Program by Class Quality Indieators 





Coefficient 


Standard Error 


t 


M=0, G=0 








Intereept 


0.58 


0.03 


12.28*** 


Direet effeet 


-0.05 


0.05 


-1. 18 


M=0, G=1 








Intereept 


0.64 


0.06 


10.16*** 


Direet effect 


-0.13 


0.05 


-2.63* 


M=1,G=0 








Intercept 


0.58 


0.06 


10.08*** 


Direct effect 


-O.OI 


0.05 


-0.14 


M=I,G=I 








Intercept 


0.59 


0.09 


6.36*** 


Direct effect 


-0.04 


0.07 


-0.56 



*p < .05; ***p < .001 
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